Method for generating image processing filter and image processing method using the image processing filter

ABSTRACT

A method for generating an image processing filter includes: adjusting; and extracting. The adjusting inputs first training image data into a neural network to generate output image data, calculates an evaluation value based on a loss function using the output image data and second training image data, and adjusts a convolution filter so as to reduce the evaluation value. The extracting extracts data from the adjusted convolution filter as data for the image processing filter. A first training image includes noise and reproduces a test pattern. A second training image includes reduced noise and reproduces the test pattern. The loss function includes a first term and a second term. The first term specifies a magnitude of a difference between the output image data and the second training image data. The second term grows smaller as symmetry of the convolution filter relative to a filter axis of symmetry increases.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority from Japanese Patent Application No.2019-180576 filed Sep. 30, 2019 and Japanese Patent Application No.2019-180577 filed Sep. 30, 2019. The entire content of each of thesepriority applications is incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to processing performed on image datausing a filter and a prediction model for processing image data.

BACKGROUND

Conventional image processing includes various types and techniques. Oneproposed image processing technique involves determining colorinformation for a background color in image data generated by an imagereading device, such as a scanner, and removing the background from theimage data in accordance with the determination results. This techniquesuppresses the appearance of off-colors in the image due to colordeviation during reading. Specifically, the technique includes colorcorrection to reduce the chroma in areas having a chroma less than orequal to a prescribed value.

Another proposed image processing technique uses filters. For example,Gaussian filters are used to perform smoothing processes, and edgeenhancement filters are used to perform edge enhancement processes.

SUMMARY

However, image processing is performed on a variety of image data. Forexample, the image data may contain various types of noise. In suchcases, the filters used to reduce noise in image data are not alwayscapable of reducing noise in some image data. Moreover, a process forreducing noise aggressively may alter the position of the image. (Forexample, the noise reduction process may cause a shift in the positionsof edge lines.)

More recently, techniques have been developed to process image datausing predictions models, such as neural networks or support vectormachines. The prediction model is trained using training image data inorder to ensure appropriate processing. However, the training image dataneeded for suitable training is not easy to prepare.

In view of the foregoing, it is an object of the present disclosure toprovide a technique capable of reducing noise while suppressingpositional changes in the image caused by image processing.

It is another object of the present disclosure to provide a technique togenerate suitable training image data for training a prediction model toprocess image data obtained through optical reading.

In order to attain the above and other objects, the present disclosureprovides a method for generating an image processing filter for imageprocessing. The method includes: adjusting; and extracting. Theadjusting adjusts a convolution filter using a neural network to acquirean adjusted convolution filter. The convolution filter includes aplurality of weights corresponding to respective ones of a plurality ofpixels within a filter region. The neural network is configured of aconvolutional layer. The convolutional layer is configured to performconvolution using the convolution filter. The adjusting includes:inputting; calculating; and second adjusting. The inputting inputs firsttraining image data into the neural network to generate output imagedata. The calculating calculates an evaluation value based on a lossfunction using the output image data and second training image data. Thesecond adjusting adjusts the plurality of weights to respective ones ofa plurality of adjusted weights so as to reduce the evaluation value.The adjusted convolution filter includes the plurality of adjustedweights. The extracting extracts data from the adjusted convolutionfilter as data for the image processing filter. The first training imagedata represents a first training image including noise. The firsttraining image reproduces a test pattern including a specific parthaving a shape linearly symmetric to an axis of symmetry. The secondtraining image data represents a second training image including reducednoise compared to the first training image. The second training imagereproduces the test pattern. The loss function includes a first term anda second term. The first term specifies a magnitude of a differencebetween the output image data and the second training image data. Thesecond tenn grows smaller as symmetry of the plurality of weightsrelative to a filter axis of symmetry in the filter region increases.The filter axis of symmetry is parallel to the axis of symmetry.

According to another aspect, the present disclosure provides an imageprocessing method using an image processing filter. The image processingmethod includes: first generating; acquiring; second generating; andstoring. The first generating generates data for the image processingfilter in accordance with the method according to the above-describedaspect. The acquiring acquires target data representing a nozzle patternby optically reading the nozzle pattern printed on a sheet. The nozzlepattern is printed on the sheet by a printing device having a printhead. The print head has a plurality of nozzles each of which configuredto eject ink. The nozzle pattern includes a plurality of lines. Theplurality of lines is parallel to each other. Each of the plurality oflines has a center axis and has a symmetric shape relative to the centeraxis. Each of the plurality of lines is printed by one of the pluralityof nozzles. The second generating generates processed image data byexecuting an image process on the target data using the image processingfilter. The storing stores the processed image data in a storage device.

According to still another aspect, the present disclosure provides amethod for generating training image data to be used for training aprediction model. The method includes: (a) generating; and (b) storing.The (a) generating generates first training image data representing afirst training image formed by arranging an object image of an object ona first background image. The first background image is at least aportion of a sheet image acquired by optically reading a sheet to beused for printing. The (b) storing stores the first training image datain a storing device. The (a) generating includes: (a1) setting; and (a2)setting. The (a1) setting sets a color value for a first object pixelwithin an object region of the first training image in which the objectimage is arranged. The color value for the first object pixel is set toa sum of a first value and a second value. The first value is obtainedby multiplying a color value for a pixel corresponding to the firstobject pixel in the object image by a first object coefficient greaterthan zero. The second value is obtained by multiplying a color value fora pixel corresponding to the first object pixel in the first backgroundimage by a first background coefficient greater than zero. The (a2)setting sets a color value for a first background pixel within anonobject region of the first training image in which the object imageis not arranged. The color value for the first background pixel is setto a value obtained by multiplying a color value for a pixelcorresponding to the first background pixel in the first backgroundimage by a coefficient greater than the first background coefficient.

According to still another aspect, the present disclosure provides anon-transitory computer readable storage medium storing a predictionmodel. The prediction model is installed on and executed by a computer.The prediction model, when executed by the computer, causes the computerto perform generating relevant data based on image data representing asheet image. The sheet image is generated by optically reading a sheeton which an object is printed. The relevant data is data about theobject. The prediction model has been trained using training image datarepresenting a training image formed by arranging an object image of anobject on a background image. The background image is at least a portionof a sheet image acquired by optically reading a sheet to be used forprinting. The training image data is generated according to a methodincluding: first setting; and second setting. The first setting sets acolor value for an object pixel within an object region of the trainingimage in which the object image is arranged. The color value for theobject pixel is set to a sum of a first value and a second value. Thefirst value is obtained by multiplying a color value for a pixelcorresponding to the object pixel in the object image by an objectcoefficient greater than zero. The second value is obtained bymultiplying a color value for a pixel corresponding to the object pixelin the background image by a background coefficient greater than zero.The second setting sets a color value for a background pixel within anonobject region of the training image in which the object image is notarranged. The color value for the background pixel is set to a valueobtained by multiplying a color value for a pixel corresponding to thebackground pixel in the background image by a coefficient greater thanthe background coefficient.

BRIEF DESCRIPTION OF THE DRAWINGS

The particular features and advantages of the disclosure as well asother objects will become apparent from the following description takenin connection with the accompanying drawings, in which:

FIG. 1 is an explanatory diagram illustrating a multifunction peripheralaccording to a first embodiment and a printing device that is separatefrom the multifunction peripheral;

FIG. 2A is an explanatory diagram illustrating a sample nozzle patternprinted on a sheet and a print head provided with a plurality of nozzlesand having no defects;

FIG. 2B is an explanatory diagram illustrating a sample nozzle patternprinted on a sheet and a print head provided with a plurality of nozzlesand having defects;

FIG. 3 is an explanatory diagram for a learning model of the firstembodiment which uses input data representing an input image of a nozzlepattern to generate output data representing an output image of the samenozzle pattern;

FIG. 4 is a flowchart illustrating steps in a training image datagenerating process executed by a processor of the multifunctionperipheral according to the first embodiment;

FIG. 5A is a flowchart illustrating steps in a standard base patternimage generating process executed by the processor of the multifunctionperipheral according to the first embodiment;

FIG. 5B is an explanatory diagram illustrating an example of a standardbase pattern image;

FIG. 6A is a flowchart illustrating steps in a nonstandard base patternimage generating process executed by the processor of the multifunctionperipheral according to the first embodiment;

FIG. 6B is an explanatory diagram illustrating an example of anonstandard base pattern image;

FIG. 7A is a flowchart illustrating steps in a background imagegenerating process executed by the processor of the multifunctionperipheral according to the first embodiment;

FIG. 7B is an explanatory diagram illustrating an example of a firstbackground image, in which two sheet images and four background imagesthat have been extracted from the sheet images are illustrated;

FIGS. 8A through 8D are explanatory diagrams for noisy pattern images,in which FIG. 8A illustrates a noisy standard pattern image generatedfrom a standard base pattern image and a first background image, FIGS.8B and 8C respectively illustrate noisy nonstandard pattern imagesgenerated from respective nonstandard base pattern images and respectivefirst background images, and FIG. 8D shows a formula for calculatingfirst target luminance values for noisy pattern images;

FIGS. 9A through 9D are explanatory diagrams for reduced-noise patternimages, in which FIG. 9A illustrates a reduced-noise standard patternimage generated from a standard base pattern image and a secondbackground image and an enhanced-contrast reduced-noise pattern imagegenerated through a contrast enhancement process, FIGS. 9B and 9Crespectively illustrate reduced-noise nonstandard pattern imagesgenerated from respective nonstandard base pattern images and respectivesecond background images and enhanced-contrast reduced-noise patternimages generated through the contrast enhancement process, and FIG. 9Dshows a formula for calculating second target luminance values forreduced-noise pattern images;

FIG. 9E is a graph showing an example of the contrast enhancementprocess executed on reduced-noise pattern image data, where thehorizontal axis represents a pre-adjusted second target luminance valueand the vertical axis represents an adjusted second target luminancevalue;

FIGS. 10A through 10C are explanatory diagrams describing a trainingprocess for the learning model, in which FIG. 10A is a flowchartillustrating steps in a training process for the learning model executedby the processor of the multifunction peripheral according to the firstembodiment, FIG. 10B shows a sample loss function used for calculatingan error value between an enhanced-contrast reduced-noise pattern imageand an output image, and FIG. 10C is an explanatory diagram illustratinga filter and a filter axis of symmetry for explaining the second term ofthe loss function shown in FIG. 10B;

FIG. 11A is a graph showing luminance values in an image of a nozzlepatter, specially illustrating a distribution of luminance values alonga single pixel line extending in a horizontal direction, where thehorizontal axis represents a position in a horizontal direction and thevertical axis represents the luminance value;

FIG. 11B is a graph showing luminance values representing a single linein the nozzle patter, in which a dashed graph line represents luminancevalues of an output image generated by the learning model prior tocompletion of training, and a solid graph line represents the luminancevalues in the reduced-noise pattern image;

FIG. 11C is a graph showing a reference case, in which a dashed graphline represents luminance values of an output image if the loss functionshown in FIG. 10B were configured only of the first term, and a solidgraph line represents the luminance values in the reduced-noise patternimage;

FIG. 11D is a graph showing the first embodiment, in which a dashedgraph line represents luminance values of an output image generated bythe learning model after training, and a solid graph line represents theluminance value in the reduced-noise pattern image;

FIG. 12 is a flowchart illustrating steps in an image process executedby the processor of the multifunction peripheral according to the firstembodiment using a convolution filter;

FIG. 13 is a block diagram illustrating a learning model of a secondembodiment;

FIG. 14 is a flowchart illustrating steps in a training image datagenerating process executed by a processor of a multifunction peripheralaccording to a third embodiment;

FIGS. 15A through 15D are explanatory diagrams of training image datagenerated in the third embodiment, in which FIGS. 15A and 15B illustratenoisy character images generated from respective character images andrespective first background images, FIGS. 15C and 15D illustratereduced-noise character images generated from respective characterimages and respective second background images and enhanced-contrastreduced-noise character images generated through the contrastenhancement process; and

FIG. 16 is a block diagram illustrating a learning model of a fourthembodiment.

DETAILED DESCRIPTION A. First Embodiment

A1. Device Configuration

FIG. 1 is an explanatory diagram illustrating a data processing deviceaccording to an embodiment of the present disclosure. A multifunctionperipheral 200 is used for data processing in the present embodiment.The multifunction peripheral 200 has a control unit 205, a printing unit260, and a reading unit 300. The control unit 205 has a processor 210, astorage device 215, a display unit 240, an operating unit 250, and acommunication interface 270. The above components are interconnected viaa bus. The storage device 215 includes a volatile storage device 220,and a nonvolatile storage device 230.

The processor 210 is a device that performs data processing, such as aCPU. The volatile storage device 220 is configured of a DRAM, forexample. The nonvolatile storage device 230 is a flash memory, forexample.

The nonvolatile storage device 230 stores a first program 231, a secondprogram 232, a third program 233, a learning model 235, and filter dataFLD. In the present embodiment, the learning model 235 is an artificialneural network based prediction model, a machine learning model that hasbeen trained through a training process described later. In the presentembodiment, the learning model 235 is a program module. The learningmodel 235 will be described later in greater detail.

Various intermediate data that the processor 210 uses for executing thefirst program 231, second program 232, third program 233, and learningmodel 235 is temporarily stored in the storage device 215 (one of thevolatile storage device 220 and nonvolatile storage device 230, forexample).

The display unit 240 is a liquid crystal display, an organic EL display,an LED display, or the like that serves to display images. The operatingunit 250 is a device that accepts user operations, such as a touchscreenarranged over the display unit 240, buttons, levers, and the like. Thecommunication interface 270 is an interface for communicating with otherdevices. For example, the communication interface 270 may be a USBinterface, a wired LAN interface, or a wireless communication interfaceconforming to the IEEE 802.11 standard.

The printing unit 260 prints images on paper (an example of the sheet)according to a prescribed method (an inkjet or laser printing method,for example).

The reading unit 300 is a reading device that optically reads anoriginal or other object using a photoelectric conversion element, suchas a CCD or CMOS. The reading unit 300 generates scan data representingthe image that was read (hereinafter referred to as the “scannedimage”). In the present embodiment, the scanned image is bitmap datarepresenting the image in grayscale. Luminance values of bitmap data inthe present embodiment are represented by 256 gradations from 0 to 255.

FIG. 1 also illustrates a printing device 100 that is separate from themultifunction peripheral 200. The printing device 100 is an inkjet typeprinting device that prints images on paper. In the present embodiment,the printing device 100 uses ink in the four colors cyan, magenta,yellow, and black. The printing device 100 is provided with a print head110 having a plurality of nozzles for ejecting ink droplets. The printhead 110 may have any of various nozzle-related defects. To identifysuch defects, the printing device 100 prints a nozzle pattern NP on asheet SH2. The sheet SH2 is a sheet of paper in the present embodiment.However, another type of sheet, such as a sheet of fabric or film, maybe used.

FIGS. 2A and 2B are explanatory diagrams illustrating a sample nozzlepattern NP. FIG. 2A illustrates a case in which the print head 110 hasno defects, while FIG. 2B illustrates a case in which the print head 110has defects. Depicted in FIGS. 2A and 2B are the nozzle pattern NPprinted on the sheet SH2, the print head 110, a plurality of nozzles Nzprovided in the print head 110, and mutually perpendicular directions D1and D2. Specifically, the drawings illustrate M number of nozzles Nz forejecting ink of one color, where M is an integer greater than or equalto two. The nozzle pattern NP is printed using the M number of nozzlesNz for the color being processed.

The first direction D1 is the scanning direction of the print head 110.The printing device 100 is provided with a reciprocating device (notillustrated) for reciprocating the print head 110 in directions alongthe first direction D1. While being moved relative to the sheet indirections along the first direction D1, the print head 110 ejects inkdroplets from the nozzles Nz to form ink dots on the sheet. The seconddirection D2 is the conveying direction of the sheet. The printingdevice 100 is provided with a conveying device (not illustrated) formoving the sheet in the second direction D2. The printing device 100prints images on a sheet by repeatedly and alternately performing ascanning process for forming ink dots while moving the print head 110and a conveying process for conveying the sheet.

The M number of nozzles Nz are arranged at different positions along thesecond direction D2 and specifically are spaced at equal intervals inthe second direction D2. The nozzle pattern NP is configured of M numberof lines PL aligned in the first direction D1. The M number of lines PLare juxtaposed in the second direction D2 orthogonal to the firstdirection D1. One line PL is printed by a single nozzle Nz in onescanning process. Each line PL has a center axis AxP indicated in thedrawings. Each line PL is shaped to be symmetrical about thecorresponding center axis AxP.

When there are no defects in the M number of nozzles (see the example ofFIG. 2A), all M number of lines PL have the same color and the samewidth in the second direction D2. Additionally, the M number of lines PLare spaced at equal intervals in the second direction D2. Hereinafter,lines PL printed by defect-free nozzles Nz will be called standardlines.

In the example of FIG. 2B, three nozzles Nza, Nzb, and Nzc have defects.The position of a first line PLa printed by the first nozzle Nzadeviates upstream in the second direction D2 from the intended position.Such positional deviations of ink dots may be caused by various factors,such as manufacturing defects of the nozzles Nz. A second line PLbprinted by the second nozzle Nzb is narrower in width than a normal linePL. In other words, the size of the ink dots is smaller. This divergencein size of ink dots may be caused by various factors, such as a defectin the nozzle driving device (piezoelectric element, heater, or thelike) used for ejecting ink droplets. A third line PLc printed by thethird nozzle Nzc has a lighter color than a normal line PL. Such colorvariation may be caused by various factors, such as ink seepage withinthe print head 110 allowing ink of two colors to mix. If theconfiguration (position, width, density, etc.) of a line PL differs fromthe configuration of a standard line in this way, the nozzle Nz thatprinted this line PL has a defect. Hereinafter, a line PL having aconfiguration that differs from the configuration of a standard linewill be called a nonstandard line.

In order to identify the configuration (position, width, density, etc.)of each line PL, the nozzle pattern NP is printed on the sheet SH2 andthe reading unit 300 (see FIG. 1) reads the sheet SH2. Subsequently, anoperator observes the scanned image of the nozzle pattern NP duringmanufacturing of the printing device 100, for example, to identify theconfiguration of each line PL and to detect defects in the nozzles Nz.Here, the cost of ink used to print the nozzle patterns NP can bereduced by printing the nozzle pattern NP at a low density. In thiscase, visibility of the nozzle pattern NP can be enhanced by performingbrightness correction (contrast enhancement, for example) of the scannedimage. However, simple brightness correction, such as correction thatinvolves adjusting the tone curve, may enhance noise in addition to thenozzle pattern NP. For example, such brightness correction may enhancefiber patterns in the paper sheet SH2 or noise attributed to digitalimage processing. The learning model 235 (see FIG. 1) is designed togenerate image data of the nozzle pattern NP by reducing noise inscanned data of the nozzle pattern NP.

FIG. 3 is an explanatory diagram for the learning model 235. Asillustrated in FIG. 3, input data 950 representing an input image 950 iof the nozzle pattern NP is inputted into the learning model 235. Theinput data 950 is obtained by executing a cropping process on thescanned data for the sheet SH2 (see FIG. 1). The cropping process cutsout the portion of the scanned data representing the nozzle pattern NP.The portion to be cut out is predetermined. As will be described later,the learning model 235 uses the input data 950 to generate output data970 representing an output image 970 i of the same nozzle pattern NP.

The input image 950 i and output image 970 i are both rectangular imageshaving two sides aligned in a horizontal direction Dx and two sidesaligned in a vertical direction Dy perpendicular to the horizontaldirection Dx. The images 950 i and 970 i are expressed with color values(luminance values in the present embodiment) for a plurality of pixelsarranged in a matrix having rows in the horizontal direction Dx andcolumns in the vertical direction Dy. The input image 950 i has apredetermined size (a pixel number IW in the horizontal direction Dx anda pixel number IH in the vertical direction Dy). The output image 970 ihas the same size as the input image 950 i. The directions D1 and D2 inFIG. 3 denote the directions D1 and D2 of the printing device 100relative to the nozzle pattern NP. The following description will assumethat the input data 950 is generated such that the first direction D1 isapproximately equivalent to the vertical direction Dy and the seconddirection D2 is approximately equivalent to the horizontal direction Dx.

The learning model 235 in the present embodiment is configured of asingle convolutional layer 310. The convolutional layer 310 executesconvolution using a single convolution filter FL. Hereinafter, theconvolution filter FL will simply be called the filter FL. Convolutionis performed using the convolutional layer 310 to calculate correlationvalues representing correlations between inputted data and the filterFL. A horizontal size Fx in FIG. 3 denotes the size (number of pixels)of the filter FL in the horizontal direction Dx, and a vertical size Fydenotes the size (number of pixels) of the filter FL in the verticaldirection Dy. In the present embodiment, Fx is an odd number five, andFy is five. The filter FL includes (Fx×Fy) number of weights W. Thenumbers i and j included in parentheses for each weight W in FIG. 3indicate a position i in the filter FL relative to the horizontaldirection Dx and a position j in the filter FL relative to the verticaldirection Dy.

From the input data 950, the convolutional layer 310 acquires a list of(Fx×Fy) number of luminance values for the (Fx×Fy) number of pixels inthe portion of the input data 950 corresponding to the position of thefilter FL. The convolutional layer 310 calculates an inner product ofthe acquired list and a list of (Fx×Fy) number of weights in the filterFL. The process of adding bias to the inner product is omitted in thepresent embodiment. The convolutional layer 310 inputs the inner productinto an activation function. The value calculated by the activationfunction denotes the correlation value. In the present embodiment, theactivation function is an identity function. By sliding the filter FL,the correlation value is calculated for the filter FL in a plurality ofpositions. The convolutional layer 310 generates bitmap datarepresenting the correlation value for each position of the filter FL.This generated bitmap data is the output data 970. The position of eachpixel in the output image 970 i represents the position of the filterFL. In the present embodiment, convolution is performed so that theoutput image 970 i has the same size (i.e., the same pixel numbers inthe horizontal direction Dx and vertical direction Dy as the input image950 i). Hence, the stride (the sliding step) is one. Zero-padding isalso performed to add pixels around the periphery of the input image 950i.

A2. Training Image Data Generating Process

FIG. 4 is a flowchart illustrating steps in a process for generatingtraining image data (training image data generating process) to be usedfor training the learning model 235. The training image data generatedin the present embodiment includes noisy data and reduced-noise data.The noisy data is image data of a nozzle pattern NP that includes noise.The reduced noise data is image data of the same nozzle pattern NP withreduced noise. In a training process described later, the noisy data isinputted into the learning model 235, causing the learning model 235 togenerate output data. The weights in the filter FL are adjusted todecrease the difference between this output data and the reduced noisedata. The trained learning model 235 (the learning model 235 aftertraining) can then use image data for a nozzle pattern NP that includesnoise to generate image data for the same nozzle pattern NP with reducednoise. In the present embodiment, the processor 210 (see FIG. 1)generates training image data according to the first program 231.

In S105 of FIG. 4, the processor 210 acquires scanned data of a sheetSH1 (see FIG. 1) by controlling the reading unit 300 to read the sheetSH1. A blank sheet on which the nozzle pattern NP has not been printedbut that can be used to print the nozzle pattern NP is used as the sheetSH1. The nozzle pattern NP may be printed on a variety of sheets,including normal and recycled paper. Moreover, the properties of sheetsmay differ among a plurality of sheets of the same type. For example,sheets among a plurality of sheets of normal paper may have differingbrightnesses. In S105 the processor 210 controls the reading unit 300 toread a plurality of sheets having different properties in order toacquire scanned data for a plurality of sheets. In the followingdescription, a scanned image represented by image data will be called asheet image. The scanned data is used as image data for a backgroundimage, as will be described later. The processor 210 stores the scanneddata acquired in S105 in the nonvolatile storage device 230.

In S110 the processor 210 selects the type of nozzle pattern NP from“standard” and “nonstandard.” “Standard” signifies a nozzle pattern NPconfigured solely of standard lines, while “nonstandard” denotes anozzle pattern NP that includes one or more nonstandard lines. Here, theprocessor 210 selects a type for which processing (the generation ofimage data) has not been completed, as will be described later. In S120the processor 210 identifies the selected type. When standard wasselected (S120: STANDARD), in S130 the processor 210 executes a processfor generating a standard base pattern image (hereinafter called astandard base pattern image generating process), and subsequentlyadvances to S150. If nonstandard was selected (S120: NONSTANDARD), inS140 the processor 210 executes a process for generating a nonstandardbase pattern image (hereinafter called a nonstandard base pattern imagegenerating process), and subsequently advances to S150.

FIG. 5A is a flowchart illustrating steps in the standard base patternimage generating process of S130 in FIG. 4. FIG. 5B is an explanatorydiagram illustrating an example of a standard base pattern image 810.The standard base pattern image 810 is an image of a nozzle pattern NPconfigured solely of standard lines that has the same size as the inputimage 950 i (see FIG. 3). In the example of FIG. 5B, the standard basepattern image 810 includes M number of lines PL0 through PLk. Thenumbers following the line symbol “PL” identify individual lines PL andare assigned in ascending order along the horizontal direction Dx, from0 to k (where k=M−1). Center axes Ax0 through Axk are the center axes ofthe corresponding lines PL0 through PLk. The center axes Ax0 through Axkextend in the vertical direction Dy. The lines PL0 through PLk arerectangular in shape and centered on the corresponding center axes Ax0through Axk so as to have linear symmetry about the corresponding centeraxes Ax0 through Axk.

In S210 of FIG. 5A, the processor 210 initializes the luminance valuesfor all pixels in the standard base pattern image 810 to white (255) andinitializes a target number j to 0.

In S215 the processor 210 sets a standard line width nW randomly(according to a random number generated) from a predetermined range ofstandard widths. The range of standard widths is the allowable range ofwidths for standard lines between a lower limit width greater than zeroand an upper limit width. The range of standard widths may be set to thedistribution range of the plurality of standard line widths in aplurality of actual input images 950 i (see FIG. 3), for example.

In S220 the processor 210 randomly sets a standard line color nD fromamong a predetermined range of standard colors. The standard color rangeis an allowable range of colors for standard lines (luminance values inthis case) that is between 0 and an upper limit luminance value. Therange of standard colors may be set to a distribution range of colorsfor a plurality of standard lines in a plurality of actual input images950 i, for example.

In S225 the processor 210 sets a standard position nS in the horizontaldirection Dx of a leftmost line. The leftmost line is the line PL0 (theline identified by the target number j=0) on the left end (i.e.,upstream end in the horizontal direction Dx) of the standard basepattern image 810 (see FIG. 5B). In the present embodiment, the standardposition nS is randomly selected from a predetermined range of standardpositions. The range of standard positions is the allowable range ofpositions for a standard line between a first position and a secondposition. The range of standard positions may be set to a distributionrange of positions for a plurality of leftmost lines in a plurality ofactual input images 950 i, for example.

In S260 the processor 210 calculates a left end position nL and a rightend position nR of a target line PLj identified by the target number j.FIG. 5B indicates the left end position nL and right end position nR forline PL0. The left end position nL is the position of the upstream endin the horizontal direction Dx, and the right end position nR is theposition of the downstream end in the horizontal direction Dx. In thepresent embodiment, M number of lines PL having the standard line widthnW are arranged at equal intervals along the horizontal direction Dx. Adifference nP of positions in the horizontal direction Dx between twoneighboring lines PL is predetermined. Thus, nL=nS+nP×j andnR=nS+nP×j+nW for the target number j.

In S265 the processor 210 fills the interior of the target line PLj inthe standard base pattern image 810 with the standard line color nD.Specifically, the processor 210 sets the luminance value for all pixelsincluded in the region from the left end position nL to the right endposition nR to the standard line color nD.

In S290 the processor 210 increments the target number j by one. In S295the processor 210 determines whether all M number of lines PL have beenprocessed. When there remain unprocessed lines PL (S295: NO), theprocessor 210 returns to S260. After processing all M number of lines PL(S295: YES), in S297 the processor 210 stores standard base patternimage data representing the standard base pattern image in the storagedevice 215 (the nonvolatile storage device 230, for example).Subsequently, the processor 210 ends the process in FIG. 5A, i.e., theprocess of S130 in FIG. 4.

FIG. 6A is a flowchart illustrating steps in the nonstandard basepattern image generating process of S140 in FIG. 4. FIG. 6B is anexplanatory diagram illustrating an example of a nonstandard basepattern image 820. The nonstandard base pattern image 820 is an image ofa nozzle pattern NP that includes nonstandard lines and has the samesize as the input image 950 i (see FIG. 3). A plurality of lines in FIG.6B are assigned the symbols PL0 through PLk, as in FIG. 5B. In theexample of FIG. 6B, the three lines PL1, PL2, and PL3 are nonstandardlines (described later in greater detail).

Steps S210, S215, S220, and S225 in FIG. 6A are identical to the samesteps in FIG. 5A. In S240 the processor 210 sets a line width pW foreach of the M number of lines PL. In the present embodiment, theprocessor 210 randomly selects a first nonstandard line from the Mnumber of lines PL. In the example illustrated in FIG. 6B, a line PL2 isselected as the first nonstandard line from among the M number of linesPL0 through PLk. The line widths pW of lines not selected as the firstnonstandard line are set to the standard line width nW. The line widthof the first nonstandard line is randomly selected from a predeterminedrange of nonstandard widths. The range of nonstandard widths is betweenzero and a nonstandard upper limit width. A width of zero indicates noline. The nonstandard upper limit width may be equivalent to the upperlimit width in the standard width range described in S215 of FIG. 5A.The width pW of the first nonstandard line PL2 in FIG. 6B differs fromthe standard line width nW.

In S245 the processor 210 sets a color pD (a luminance value in thiscase) for each of the M number of lines. In the present embodiment, theprocessor 210 randomly selects a second nonstandard line from the Mnumber of lines. In the example illustrated in FIG. 6B, a line PL3 isselected as the second nonstandard line from among the M number of linesPL0 through PLk. The color pD for lines not selected as the secondnonstandard line are set to the standard line color nD. The color pD ofthe second nonstandard line is randomly selected from a predeterminedrange of nonstandard colors. The range of nonstandard colors is between0 and a nonstandard upper limit luminance value. The nonstandard upperlimit luminance value may be the same as or greater than the upper limitluminance value for the standard color range described in S220 of FIG.5A. The color pD of the second nonstandard line PL3 in FIG. 6B differsfrom the standard line color nD.

In S250 the processor 210 sets a positional deviation pZ in thehorizontal direction Dx for each of the M lines. In the presentembodiment, the processor 210 randomly selects a third nonstandard linefrom the M number of lines. In the example illustrated in FIG. 6B, aline PL1 is selected as the third nonstandard line from among the Mnumber of lines PL0 through PLk. The positional deviation pZ for linesnot selected as the third nonstandard line is set to zero. Thepositional deviation pZ for the third nonstandard line is randomlyselected from a predetermined range of deviations. The range ofdeviations is between −XB and +XB. The value XB is predetermined throughexperimentation. The positional deviation pZ for the third nonstandardline PL1 in FIG. 6B is not zero.

In S260 a the processor 210 calculates the left end position nL andright end position nR of the target line PLj identified by the targetnumber j. This step differs from S260 in FIG. 5A in that the line widthpW(j) and the positional deviation pZ(j) for the target number j areconsidered. Specifically, the processor 210 uses the calculationsnL=nS+nP×j+pZ(j) and nR=nS+nP×j+pW(j)+pZ(j). In the example of FIG. 6B,as stated above, the position of the third nonstandard line PL1 in thehorizontal direction Dx is offset according to the positional deviationpZ. Therefore, the left end portion nL and right end portion nR of thethird nonstandard line PL1 are deviated. Further, the width of the firstnonstandard line PL2 is set to the width pW, which differs from thestandard line width nW. Therefore, the right end position nR of thefirst nonstandard line PL2 is deviated.

In S265 a the processor 210 fills the interior of the target line PLj inthe nonstandard base pattern image 820 with the color pD(j) for thetarget line PLj. Specifically, the luminance value of all pixels in therange from the left end position nL to the right end position nR are setto the color pD(j) for the target line PLj.

Note that nonstandard lines are independently selected in each of stepsS240, S245, and S250. Hence, two or more of the three parameters widthpW, color pD, and positional deviation pZ for a single nonstandard linemay differ from the corresponding parameters of the standard line.

Subsequently, the processor 210 executes the same steps S290 and S295described in FIG. 5A. When there remain unprocessed lines PL (S295: NO),the processor 210 returns to S260 a. When all lines PL have beenprocessed (S295: YES), in S297 a the processor 210 stores nonstandardbase pattern image data representing the nonstandard base pattern imagein the storage device 215 (the nonvolatile storage device 230, forexample). Subsequently, the processor 210 ends the process of FIG. 6A,i.e., the process in S140 of FIG. 4.

In S150 of FIG. 4, the processor 210 executes a smoothing process on thebase pattern image data generated in S130 or S140 to generate smoothedbase pattern image data. The smoothing process is also called a blurringprocess. Outlines of the nozzle pattern NP are blurred in the actualscanned image of the nozzle pattern NP. In this process, S150 isperformed on the base pattern image data so that the base pattern imageapproximates to the actual scanned image. A mean filter is used in thesmoothing process of the present embodiment. The smoothing process maybe performed using various other processes, such as a process using amedian filter, a Gaussian filter, or another smoothing filter.

In S155 the processor 210 executes a process for generating a backgroundimage (hereinafter called a background image generating process). FIG.7A is a flowchart illustrating steps in the background image generatingprocess. In S310 of FIG. 7A, the processor 210 randomly selects onesheet image from a plurality of sheet images for a plurality of sheets(S105 of FIG. 4) to be a target sheet image. In S315 the processor 210extracts a background image from the target sheet image. The backgroundimage is a portion of the target sheet image having the same size as theinput image 950 i (see FIG. 3). The position of the background image inthe target sheet image is set randomly. Hereinafter, the backgroundimage extracted from the sheet image will also be called the firstbackground image, and image data for the first background image will becalled first background image data.

FIG. 7B is an example of the first background image. The example in FIG.7B indicates two sheet images SHR1 and SHR2, and background images BG1through BG4 that have been extracted from the sheet images SHR1 andSHR2. One background image is extracted through each execution of theprocess in FIG. 7A.

In S160 of FIG. 4, the processor 210 generates noisy pattern image datarepresenting a noisy pattern image. The noisy pattern image is generatedusing the base pattern image generated in S130 or S140 and the firstbackground image. FIGS. 8A through 8D are explanatory diagrams of noisypattern images. FIG. 8A illustrates a noisy standard pattern image 850 agenerated from a standard base pattern image 810 a and a firstbackground image 830 a. FIGS. 8B and 8C respectively illustrate noisynonstandard pattern images 850 b and 850 c generated from respectivenonstandard base pattern images 820 b and 820 c and respective firstbackground images 830 b and 830 c. A luminance value Pt(x, y) in thedrawings denotes the luminance value of a pixel (x, y) in thecorresponding base pattern images, i.e., the standard base pattern image810 a and the nonstandard base pattern images 820 b and 820 c(hereinafter called a base pattern luminance value Pt). The value xdenotes the pixel position in the image relative to the horizontaldirection Dx, and the value y denotes the pixel position in the imagerelative to the vertical direction Dy. For luminance values Pu and Pn ofother images, (x, y) indicates the position of a pixel in the image. Theluminance value Pu is the luminance value of a pixel in the firstbackground images 830 a through 830 c (hereinafter called the firstbackground luminance value Pu). The luminance value Pn is a luminancevalue of a pixel in the noisy pattern images, i.e., the noisy standardpattern image 850 a and the noisy nonstandard pattern images 850 b and850 c (hereinafter called the first target luminance value Pn).

FIG. 8D shows a formula for calculating the first target luminance valuePn. When the base pattern luminance value Pt(x, y) is 255, i.e., whenpixel (x, y) represents the background and not a line PL, the firsttarget luminance value Pn(x, y) is set to the same value as the firstbackground luminance value Pu(x, y). In other words, the weight of thefirst background luminance value Pu is one and the weight of the basepattern luminance value Pt is zero.

When the base pattern luminance value Pt(x, y) is less than 255, i.e.,when the pixel (x, y) represents a line PL and not the background, Pn(x,y) is set to k1×Pu(x, y)+k2×Pt(x, y). Weights k1 and k2 are both greaterthan zero. In the present embodiment, k1+k2=1. For example, k1=0.75 andk2=0.25.

The processor 210 sets the luminance value for each pixel in the noisypattern image according to the formula in FIG. 8D. To calculate thefirst target luminance value Pn of pixels representing a line PL, thefirst background luminance value Pu is used in addition to the basepattern luminance value Pt. Accordingly, the noisy pattern image cansuitably reproduce the sheet image, i.e., the scanned image of a sheeton which the base pattern image was printed.

In S165 of FIG. 4, the processor 210 stores the noisy pattern image datagenerated in S160 in the storage device 215 (the nonvolatile storagedevice 230 in the present embodiment).

In S170 the processor 210 identifies a representative color value in thefirst background image generated in S155. In the present embodiment, theprocessor 210 calculates the representative value to be the averageluminance value of all pixels in the first background image.

In S175 the processor 210 generates reduced-noise pattern image datarepresenting a reduced-noise pattern image. The reduced-noise patternimage is generated using the base pattern image generated in S130 orS140 and a second background image represented by the representativecolor value identified in S170. FIGS. 9A through 9D are explanatorydiagrams for reduced-noise pattern images. FIG. 9A illustrates areduced-noise standard pattern image 851 a generated from the standardbase pattern image 810 a and a second background image 830 am. FIGS. 9Band 9C illustrate respective reduced-noise nonstandard pattern images851 b and 851 c generated from the respective nonstandard base patternimages 820 b and 820 c and respective second background images 830 bmand 830 cm. The color values of all pixels in the second backgroundimages 830 am through 830 cm are set to the representative color value.A second background luminance value Pum(x, y) is the luminance value ofa pixel (x, y) in the second background images 830 am through 830 cm andis equivalent to the representative color value. A luminance value Pc(x,y) is the luminance value of a pixel (x, y) in the reduced-noise patternimages, i.e., the reduced-noise standard pattern image 851 a and thereduced-noise nonstandard pattern images 851 b and 851 c (hereinaftercalled a second target luminance value Pc).

FIG. 9D shows a formula for calculating the second target luminancevalue Pc. When the base pattern luminance value Pt(x, y) is 255, i.e.,when the pixel (x, y) represents the background and not a line PL, thesecond target luminance value Pc(x, y) is set to the same value as thesecond background luminance value Pum(x, y), i.e., the representativecolor value (the average value of first background luminance values Puin the present embodiment). Thus, the weight of the second backgroundluminance value Pum is one and the weight of the base pattern luminancevalue Pt is zero.

When the base pattern luminance value Pt(x, y) is smaller than 255,i.e., when the pixel (x, y) represents a line PL and not the background,the second target luminance value Pc(x, y) is set to k1×Pum(x,y)+k2×Pt(x, y). The weight k1 of the second background luminance valuePum is equivalent to the weight k1 of the first background luminancevalue Pu in FIG. 8D. The weight k2 of the base pattern luminance valuePt is equivalent to the weight k2 of the base pattern luminance value Ptin FIG. 8D.

The processor 210 sets the luminance value for each pixel in thereduced-noise pattern image according to the formula in FIG. 9D. Tocalculate the second target luminance value Pc of pixels representing aline PL, the second background luminance value Pum (i.e., therepresentative color value) is used in addition to the base patternluminance value Pt. Accordingly, the reduced-noise pattern image cansuitably represent an image formed by reducing noise (fiber patterns inthe sheet, for example) in the sheet image, i.e., the scanned image of asheet on which the base pattern image was printed.

In S180 of FIG. 4, the processor 210 executes a contrast enhancementprocess on the reduced-noise pattern image data. FIG. 9E is a graphshowing an example of the contrast enhancement process, where thehorizontal axis represents the pre-adjusted second target luminancevalue Pc and the vertical axis represents an adjusted second targetluminance value Pee. The correlations in the graph are configured suchthat the adjusted second target luminance value Pee changes from 0 to255 as the pre-adjusted second target luminance value Pc changes from afirst threshold Va to a second threshold Vb (0<Va<Vb<255). The adjustedsecond target luminance value Pcc is set to 0 when Pc<Va and is set to255 when Vb<Pc. Note that various other processes for enhancing contrastmay be executed in place of the process based on the correlations inFIG. 9E.

FIGS. 9A through 9C illustrate enhanced-contrast reduced-noise patternimages 860 a through 860 c. Owing to this enhanced contrast, the linesPL and background can be easily distinguished from each other. Imagedata representing the enhanced-contrast reduced-noise pattern images 860a through 860 c will be called adjusted reduced-noise pattern imagedata.

In S185 of FIG. 4, the processor 210 stores the adjusted (i.e.,enhanced-contrast) reduced-noise pattern image data in the storagedevice 215 (the nonvolatile storage device 230 in the presentembodiment) in association with the noisy pattern image data.

In S190 the processor 210 determines whether the generation of imagedata is complete. In the present embodiment, the condition forcompletion requires that the number of times image data has beengenerated for a standard nozzle pattern NP be greater than or equal to apredetermined standard number P (where P is an integer greater than orequal to two) and that the number of times image data has been generatedfor a nonstandard nozzle pattern NP be greater than or equal to apredetermined nonstandard number Q (where Q is an integer greater thanor equal to two). Note that other conditions for completion may be used.

When the generation of image data is not complete (S190: NO), theprocessor 210 returns to S110. When the processor 210 determines thatthe generation of image data is complete (S190: YES), the processor 210ends the process of FIG. 4. At this time, a plurality of sets of noisypattern image data and adjusted reduced-noise pattern image data isstored in the nonvolatile storage device 230.

As described above, noisy pattern image data is generated in S155 andS160 of FIG. 4 in the training image data generating process accordingto the present embodiment. As described with reference to FIGS. 8Athrough 8C, the noisy pattern images 850 a through 850 c are formed byarranging the nozzle pattern NP (a plurality of lines PL) on thecorresponding first background images 830 a through 830 c. As describedin S160, a noisy pattern image is generated using the first backgroundimage and the base pattern image. As described in S105 and S155, thefirst background image is at least a portion of the scanned image (i.e.,the sheet image) acquired by optically reading the sheet SH1 to be usedfor printing. As described in S130 and S140, the base pattern image isan image of the nozzle pattern NP. In S165 the noisy pattern image datais stored in the storage device 215.

As described in FIG. 8D, the process for setting color values (luminancevalues in this case) for pixels in the noisy pattern image includes twoprocesses. The first process is performed for cases in which the targetpixel represents the nozzle pattern NP (see the bottom line of FIG. 8D).In the first process, the luminance value (first target luminance value)Pn(x, y) for the target pixel (x, y) is set to the weighted sum of thefirst background luminance value Pu(x, y) and the base pattern luminancevalue Pt(x,y) for the target pixel (x, y). The weight k1 multiplied bythe first background luminance value Pu is greater than zero, and theweight k2 multiplied by the base pattern luminance value Pt is greaterthan zero. The second process is performed for cases in which the targetpixel represents a portion where the nozzle pattern NP is not present(see the top line of FIG. 8D). In the second process, the luminancevalue (first target luminance value) Pn(x, y) for the target pixel (x,y) is set to a value acquired by multiplying a weight larger than theweight k1 of the first background luminance value Pu used in the firstprocess (one in the present embodiment) by the first backgroundluminance value Pu(x, y) for the target pixel (x, y). Through the aboveprocesses, the noisy pattern image can suitably reproduce the imageobtained by optically reading a sheet on which the nozzle pattern NP wasprinted. The learning model 235 is used to process actual image dataacquired by optically reading a sheet on which the nozzle pattern NP wasprinted. The process according to the embodiment illustrated in FIG. 4can suitably generate training image data for training this type oflearning model 235.

In S170 the processor 210 identifies a representative color value thatrepresents a plurality of luminance values associated with a pluralityof positions in the background image that was generated in S155. In S175the processor 210 generates reduced-noise pattern image data and in S180adjusts the contrast of the reduced-noise pattern image data. Asdescribed with reference to FIGS. 9A through 9C, the reduced-noisepattern images 851 a through 851 c are images formed by arranging thenozzle pattern NP (a plurality of lines PL) over the correspondingsecond background images 830 am through 830 cm. As described in S175, areduced-noise pattern image is generated using a second background imageand the base pattern image. As described in S170 and S175, the secondbackground image is an image represented by the representative colorvalue. In S185 the reduced-noise pattern image data adjusted in S180 isstored in the storage device 215.

As described in FIG. 9D, the process for setting color values (luminancevalues in this case) for pixels in the reduced-noise pattern imageincludes two processes. The first process is performed for cases inwhich the target pixel represents the nozzle pattern NP (see the bottomline of FIG. 9D). In the first process, the luminance value (secondtarget luminance value) Pc(x, y) for the target pixel (x, y) is set tothe weighted sum of the second background luminance value Pum(x, y) andthe base pattern luminance value Pt(x, y) for the target pixel (x, y).The weight k1 multiplied by the second background luminance value Pum isgreater than zero, and the weight k2 multiplied by the base patternluminance value Pt is greater than zero. The second process is performedfor cases in which the target pixel represents a portion where thenozzle pattern NP is not present (see the top line of FIG. 9D). In thesecond process, the luminance value (second target luminance value)Pc(x, y) for the target pixel (x, y) is set to a value obtained bymultiplying a weight larger than the weight k1 of the second backgroundluminance value Pum used in the first process (one in the presentembodiment) by the second background luminance value Pum(x, y) for thetarget pixel (x, y). Through the above processes, the reduced-noisepattern image can suitably represent an image obtained by reducing noisein a noisy pattern image. The process according to the embodiment inFIG. 4 can suitably generate training image data for training using animage that includes noise and an image with reduced noise.

As described in S170, the representative color value in the presentembodiment is the average value of a plurality of color values for aplurality of pixels in the first background image. Hence, since thesecond background image can suitably represent a background image whosenoise has been reduced in comparison to the first background image, thisprocess can generate suitable pattern image data with reduced noise.Note that the representative color value is not limited to an averagevalue but may be any of various values identified using a plurality ofcolor values, such as the median or mode. Further, the representativecolor value may be identified using some of a plurality of pixelsselected from uniformly distributed positions in the first backgroundimage rather than all pixels in the first background image.

As described with reference to FIG. 5B and the like, the nozzle patternNP includes M number of lines PL aligned in the first direction D1 andjuxtaposed in the second direction D2 orthogonal to the first directionD1 (where M is an integer greater than or equal to two). Thus, thepresent embodiment can suitably generate training image data for thistype of nozzle pattern NP.

As described in S190, a noisy pattern image and a reduced-noise patternimage for nonstandard nozzle pattern NP are generated Q number of times(where Q is an integer greater than or equal to two). As described inFIG. 6A, the standard line width nW, standard line color nD, standardposition nS, and positional deviation pZ of zero are associated with astandard line included in the noisy pattern image. The standard linewidth nW, standard line color nD, and standard position nS areassociated with the noisy pattern image and are set for each noisypattern image. As indicated in S260 a of FIG. 6A, the position of thestandard line in the horizontal direction Dx is set according to thestandard position nS and the target number j (i.e., the number of theline PL in the horizontal direction Dx). As described with reference toFIGS. 6A, 6B, and the like, each of the Q number of noisy pattern imagesincludes one or more nonstandard lines. A nonstandard line has one ormore of the following three parameter values: (1) a color pD (luminancein this case) that differs from the standard line color nD; (2) aposition that differs from the standard position in the horizontaldirection Dx corresponding to the standard position nS and target numberj; and (3) a width pW that differs from the standard line width nW.Accordingly, the third embodiment can suitably generate training datafor training the learning model 235, which is used to process images ofnozzle patterns having defects, such as those illustrated in FIG. 2B.

Through the process in FIG. 6A, the Q number of noisy nonstandardpattern images include the following three types of noisy patternimages: (1) a noisy pattern image that includes one or more nonstandardlines having a color pD (luminance in this case) that differs from thestandard line color nD; (2) a noisy pattern image that includes one ormore nonstandard lines arranged at different positions from the standardpositions in the horizontal direction Dx; and (3) a noisy pattern imagethat includes one or more nonstandard lines having a width pW thatdiffers from the standard line width nW. Hence, suitable training imagedata can be generated for use in training when the luminance, positionin the horizontal direction Dx, and width may be distorted. For example,the noisy nonstandard pattern image 850 b in FIG. 8B may include one ormore nonstandard lines having a color pD that differs from the standardline color nD, and one or more nonstandard lines arranged at positionsin the horizontal direction Dx differing from the standard positions.The noisy nonstandard pattern image 850 c in FIG. 8C may include one ormore nonstandard lines having a width pW that differs from the standardline width nW. In this way, a single noisy pattern image may include aplurality of types of nonstandard lines.

In the process of FIG. 6A, nonstandard lines are randomly selected foreach noisy pattern image. Therefore, the total number of nonstandardlines may differ between a plurality of noisy nonstandard patternimages. Hence, training image data can be suitably generated when thetotal number of nonstandard lines included in the nozzle pattern NP canvary. For example, the total number of nonstandard lines may bedifferent between the noisy nonstandard pattern image 850 b in FIG. 8Band the noisy nonstandard pattern image 850 c in FIG. 8C.

A3. Training Process

FIGS. 10A through 10C are explanatory diagrams describing a trainingprocess for the learning model 235. FIG. 10A is a flowchart illustratingsteps in a sample training process for the learning model 235. In thepresent embodiment, the learning model 234 is trained so that, whenimage data of a nozzle pattern NP is inputted therein, the learningmodel 235 generates image data for the same nozzle pattern NP but withreduced noise. This training adjusts the (Fx×Fy) number of individualweights in the filter FL (see FIG. 3). Below, the training processexecuted by the multifunction peripheral 200 (see FIG. 1) will bedescribed. The processor 210 performs training according to the secondprogram 232. Note that training may be executed by another dataprocessing device having a high computing power.

In S310 of FIG. 10A, the processor 210 initializes a plurality ofoperation parameters for the learning model 235 (the (Fx×Fy) number ofweights in the filter FL in this case). For example, each of theoperation parameters is set to a random number.

In S315 the processor 210 selects V number of sets of noisy patternimage data (where V is an integer greater than or equal to one) from theplurality of sets of noisy pattern image data, inputs the V number ofsets of data into the learning model 235, and generates V number of setsof output image data. Here, unused sets of noisy pattern image data maybe selected from the plurality of sets of noisy pattern image data asthe V number of sets of noisy pattern image data. Additionally, the Vnumber of sets of noisy pattern image data may be randomly selected fromthe plurality of sets of data.

In S320 the processor 210 calculates an error value for each of the Vnumber of sets of noisy pattern image data inputted into the learningmodel 235. The error value is an evaluation value on the differencebetween the adjusted reduced-noise pattern image data and the outputimage data. The error value is calculated according to a predeterminedloss function. FIG. 10B is an explanatory diagram showing a sample lossfunction. In the present embodiment, a loss function LF is the sum of afirst term Limg and a second term Lfilter.

The first term Limg denotes the difference between the adjustedreduced-noise pattern image and the output image represented by outputimage data. In the present embodiment, the first term Limg is theroot-mean-square error normalized to a value between 0 and 255. Thepixel number IW in the first term Limg in FIG. 10B denotes the number ofpixels in the horizontal direction Dx of each image, while the pixelnumber IH denotes the number of pixels in the vertical direction Dy ofeach image (see FIG. 3). A pixel value Po(x, y) denotes the value of apixel (x, y) in the output image, where the value of x indicates theposition of the pixel in the image relative to the horizontal directionDx and the value y indicates the position of the pixel in the imagerelative to the vertical direction Dy. The pixel value Pcc(x, y) denotesthe value of the pixel (x, y) in the adjusted reduced-noise patternimage (see FIGS. 9A through 9C). The first term Limg becomes smaller asthe difference between the adjusted reduced-noise pattern image and theoutput image grows smaller. Note that the first term Limg may beexpressed by various functions that calculate a smaller value as thedifference between two images grows smaller.

The second term Lfilter is an evaluation value indicating the degree ofsymmetry of weights Win the filter FL. FIG. 10C is an explanatorydiagram of the second term Lfilter. This diagram illustrates a filter FLand a filter axis of symmetry AxF. The filter axis of symmetry AxF isparallel to the axis of symmetry in each line PL (the center axis AxPand center axes Ax0 through Axk, for example) in the nozzle pattern NP(see FIGS. 2A, 5B, 6B, etc.). In the present embodiment, the filter axisof symmetry AxF is aligned in the first direction D1 (i.e., the verticaldirection Dy). The filter axis of symmetry AxF also passes through thecenter of the filter FL in the second direction (i.e., the horizontaldirection Dx). Since the horizontal size Fx of the filter FL is fivewhich is an odd number in the present embodiment, the filter axis ofsymmetry AxF overlaps weights W(3, 1) through W(3, 5) in the thirdcolumn.

The second term Lfilter is the average of the absolute values of thedifferences between two weights W arranged at symmetrical positionsabout the filter axis of symmetry AxF. Symmetrical pairs PW1 and PW2 inFIG. 10C are examples of two weights W arranged at symmetricalpositions. The first pair PW1 is configured of weights W(1, 2) and W(5,2). The second pair PW2 is configured of weights W(2, 4) and W(4, 4).

In the formula for the second term Lfilter shown in FIG. 10B, the lowerbounds and upper bounds of the indices i and j in the summation notationare set such that all weights on the left side of the filter axis ofsymmetry AxF are selected. In the example of the embodiment, ten weightsW(1, 1) through W(1, 5) and W(2, 1) through W(2, 5) are selected. Thevariable added through the summation notation is the absolute value ofthe difference between symmetrical weight pair (W(i, j) and W(Fx+1−i,j)) identified by the indices i and j.

The second term Lfilter grows smaller as the symmetry of the pluralityof weights W about the filter axis of symmetry AxF increases. A secondterm. Lfilter of this type is included in the loss function LF in orderto suppress distortion (positional deviation, for example) in the linePL caused by image processing using the filter FL.

FIGS. 11A through 11D are explanatory diagrams illustrating thepositions of lines PL. FIG. 11A is a graph showing the luminance valuesin an image of the nozzle pattern NP, where the horizontal axisrepresents a position Px in the horizontal direction Dx and the verticalaxis represents the luminance value V. This graph illustrates thedistribution of luminance values V along a single pixel line extendingin the horizontal direction Dx. The direction of the vertical axis isdownward. Hence, the luminance value V is smaller in the upper portionof the graph and larger in the lower portion of the graph. Each peak inthe graph represents a single line PL, and each valley represents abackground area BG. The variables for the vertical and horizontal axesare common for all graphs in FIGS. 11A through 11D.

FIG. 11B is a graph of luminance values representing a single line PL.The dashed graph line indicates a luminance value Po of the output imagegenerated by the learning model 235, and the solid graph line representsa luminance value Pc in a reduced-noise pattern image. FIG. 11Bindicates the luminance value Po outputted by the learning model 235prior to completion of training. As with the pixel value Po(x, y) forthe output image described in FIG. 10B, the luminance value Po indicatesthe luminance value at each pixel position in the output image. As withthe luminance value Pc(x, y) described in FIG. 9A and the like, theluminance value Pc indicates the luminance value at each pixel positionin the reduced-noise pattern image. As shown in FIG. 11B, a differenceis produced between these luminance values Po and Pc. Training in thepresent embodiment reduces the difference between the luminance valuesPo and Pcc rather than the difference between the luminance values Poand Pc. (As with the pixel value Pcc described in FIG. 10B and the like,the luminance value Pcc indicates the luminance value at each pixelposition in the adjusted reduced-noise pattern image.) However, tosimplify the description, this description will assume that trainingreduces the difference between the luminance values Po and Pc.

FIG. 11C is a graph illustrating a reference case. This graph indicatesthe luminance values Po and Pc if the loss function LF were configuredonly of the first term Limg. The luminance value Po is the luminancevalue outputted by the learning model 235 after training. A first peakposition PxC in FIG. 11C denotes the position of a peak in the line PLrepresented by the luminance values Pc in the reduced-noise patternimage. This first peak position PxC is equivalent to the position of theaxis of symmetry for the line PL and is identical to the peak positionof the line PL in the noisy pattern image inputted into the learningmodel 235. Hereinafter, the first peak position PxC will also be calledthe axis of symmetry AxC.

A second peak position PxO in FIG. 11 indicates the position of a peakin the line PL represented by the luminance values Po in the outputimage. The second peak position PxO deviates from the first peakposition PxC for the following reason. During training using the firstterm Limg, the filter FL is adjusted to reduce the error valuecalculated according to the first term Limg without discriminatingbetween the line PL and background area BG. As a result, in order toreduce the large difference with the background area BG (see FIG. 11B),the adjusted filter FL may adjust the difference between the luminancevalues Po and Pc in areas representing the line PL (see FIG. 11C)asymmetrically to the axis of symmetry AxC of the line PL, therebydistorting the line PL in the output image. For example, the second peakposition PxO in the line PL of the output image may deviate from theoriginal first peak position PxC. If this output image were used, alarge error would be identified at the position of the line PL in theoutput image.

FIG. 11D is an explanatory diagram for the present embodiment. Theluminance value Po in this graph is the luminance value outputted by thelearning model 235 after training. As shown in the graph, the peakposition PxO of the line PL represented by the luminance value Po in theoutput image is approximately equivalent to the original peak positionPxC for the following reason. The loss function LF (see FIG. 10B)includes the second term Lfilter in addition to the first term Limg. Thevalue of the second term. Lfilter increases as the symmetry of weights Win the filter FL decreases. Through operations with the convolutionallayer 310 (see FIG. 3), the filter FL is applied to a plurality ofpositions distributed uniformly through the input image (all pixelpositions in the input image in the present embodiment). In other words,the filter FL is applied to a plurality of positions overlapping oneline PL that have linear symmetry to the axis of symmetry AxC of theline PL. Hence, if the filter FL were to adjust the difference betweenthe luminance values Po and Pc asymmetrically relative to the axis ofsymmetry AxC of the line PL, as in the example of FIG. 11C, symmetry ofthe weights W relative to the filter axis of symmetry AxF (see FIG. 10C)would be low. As a result, although the value of the first term Limg canbe decreased, the value of the second term Lfilter would increase,causing an increase in the error values based on the loss function LF.

When weights W in the filter FL are adjusted to reduce error valuesbased on the loss function LF (see FIG. 10B), symmetry of weights Wrelative to the filter axis of symmetry AxF (see FIG. 10C) is increased.Therefore, the adjusted filter FL adjusts the difference between theluminance values Po and Pee symmetrically relative to the axis ofsymmetry AxC of the line PL, thereby suppressing deviations between thesecond peak position PxO in the line PL of the output image and theoriginal peak position PxC.

In S325 of FIG. 10A, the processor 210 uses the V number of error valuesbased on this loss function LF to adjust the plurality of operationparameters in the learning model 235 (the weights W of the filter FL inthe present embodiment). Specifically, the processor 210 adjusts theplurality of operation parameters according to a predetermined algorithmin order to reduce the error values. For example, the algorithm may beone using a backpropagation method and a gradient descent method.

In S330 the processor 210 determines whether training is complete.Training is determined to be complete on the condition that all errorvalues calculated in S320 are smaller than a predetermined errorthreshold, for example. Alternatively, training may be determined to becomplete on the condition that the process in S315 through S325 has beenexecuted at least a predetermined count threshold.

If the processor 210 determines that training is not complete (S330:NO), the processor 210 returns to S315. When determining that trainingis complete (S330: YES), in S335 the processor 210 extracts data for thefilter FL from the learning model 235 (i.e., data on the (Fx×Fy) numberof weights W). In S340 the processor 210 stores filter data FLDrepresenting the data for the filter FL in the storage device 215 (thenonvolatile storage device 230 in the present embodiment), andsubsequently ends the process of FIG. 10A.

A4. Image Process

FIG. 12 is a flowchart illustrating steps in an image process using thefilter data FLD (see FIG. 1). In this image process, a filtering processusing the filter FL is performed on scanned data for the sheet SH2 onwhich the nozzle pattern NP has been printed. The image process isperformed to check for defects in nozzles Nz of the print head 110. Theprocess of FIG. 12 may be performed when manufacturing the printingdevice 100, for example. In the present embodiment, the processor 210 ofthe multifunction peripheral 200 executes the image process according tothe third program 233.

In S410 of the image process, the printing device 100 prints the nozzlepattern NP on the sheet SH2. For example, an operator performsoperations on the printing device 100 for controlling the printingdevice 100 to print the nozzle pattern NP. Alternatively, the printingdevice 100 may be connected to the multifunction peripheral 200, and theprocessor 210 may issue a print command to the printing device 100.

In S420 the sheet SH2 on which the nozzle pattern NP has been printed isoptically read by the reading unit 300 to generate scanned data. In S430the processor 210 acquires the scanned data from the reading unit 300and executes a cropping process on the scanned data to cut out theportion of the data representing the nozzle pattern NP. In this way, theprocessor 210 obtains the target data. The portion of the scanned imagethat is cut out is predetermined. Note that the cropping process may beomitted, and the scanned data may be used as the target data as is.

In S440 the processor 210 executes a filtering process on the targetdata using the filter FL to generate processed image data. The filteringprocess is identical to the convolution process performed by theconvolutional layer 310 (see FIG. 3). The processed image represented bythe processed image data represents an undistorted nozzle pattern NPthat is identical to the nozzle pattern NP in the target data. Thus,this process reduces noise and enhances contrast in the processed image.

In S450 the processor 210 stores the processed image data in the storagedevice 215 (the nonvolatile storage device 230 in the presentembodiment). In S460 the processor 210 analyzes the processed image datato determine whether the nozzle pattern NP in the processed image hasdefects. For example, the processor 210 detects a plurality of lines PLin the processed image according to a well-known line detection method(edge extraction and Hough transform, for example). The processor 210identifies the width, luminance value, and position in the horizontaldirection Dx of each line PL by analyzing the plurality of detectedlines PL. If even one parameter from among the position, width, andluminance value for one target line falls outside an allowable range forthat parameter, the processor 210 determines that the target line has adefect. However, if all parameters for position, width, and luminancevalue are within the allowable ranges for the target line, the processor210 determines that the target line has no defects. In S470 theprocessor 210 stores data indicating these determination results in thestorage device 215 (the nonvolatile storage device 230 in this example),and subsequently ends the process of FIG. 12.

As described above, a filter FL for image processing is generated in theprocess of FIG. 10A. In S315 the processor 210 generates output imagedata by inputting noisy pattern image data into the learning model 235.As described in FIG. 3, the learning model 235 is a neural network basedmodel configured of a single convolutional layer 310. The filterFL(convolution filter FL) has a weight W for each pixel within a filterregion, which is the region of the inputted image corresponding to thefilter FL. The convolutional layer 310 performs convolution using thefilter FL. In S320 of FIG. 10A, the processor 210 uses the output imagedata and reduced-noise pattern image data to calculate error valuesbased on the loss function LF. In S325 the processor 210 adjusts theweights W in the filter FL to reduce these error values. In S335 theprocessor 210 extracts filter data FLD from the filter FL indicating theadjusted weights W as filter data for image processing. As illustratedin FIGS. 8A through 8C, the noisy pattern images 850 a through 850 c areimages including noise and reproduce the nozzle pattern NP that includeslines PL. As described in FIG. 5B, the lines PL0 through PLk havesymmetric shapes relative to their corresponding center axes Ax0 throughAxk. As described in FIG. 9A through 9C, the enhanced-contrastreduced-noise pattern images 860 a through 860 c are images of thenozzle pattern NP having reduced noise compared to the noisy patternimages 850 a through 850 c. The enhanced-contrast reduced-noise patternimages 860 a through 860 c represent the same nozzle pattern NP as thenozzle pattern NP in the noisy pattern images 850 a through 850 c. Asshown in FIG. 10B, the loss function LF includes the first term Limg andthe second term Lfilter. The first term. Limg specifies the magnitude ofdifference between the output image data and the reduced-noise patternimage data. The second term Lfilter grows smaller as symmetry of theweights W relative to the filter axis of symmetry AxF in the filter FL(see FIG. 10C) increases. The filter axis of symmetry AxF is parallel tothe center axes Ax0 through Axk of the corresponding lines PL0 throughPLk. Accordingly, an image process performed on an image of the nozzlepattern NP using the filter FL can reduce noise and suppress distortionin the nozzle pattern NP caused by image processing. Specifically, thisprocess suppresses changes in the positions of lines PL, as describedwith reference to FIGS. 11A through 11D.

Both image data for standard nozzle patterns NP and image data fornonstandard nozzle patterns NP are used in training. Accordingly, thelearning model 235 (and thus the filter FL) can generate an output imageof the same undistorted nozzle pattern NP, irrespective of whether thenozzle pattern NP in the scanned image includes nonstandard lines.

As described in S180 and S185 of FIG. 4 and in FIGS. 9A through 9C, theenhanced-contrast reduced-noise pattern images 860 a through 860 c usedfor calculating error values have more enhanced contrast than the noisypattern images 850 a through 850 c (see FIGS. 8A through 8C). Therefore,an image process using the filter FL can enhance contrast.

As described in FIGS. 10B and 10C, the second term Lfilter is smallerfor a smaller difference between two weights arranged in the filter FLat symmetric positions about the filter axis of symmetry AxF. Thus, thesecond term Lfilter can suitably suppress distortion (positionaldeviation, for example) of lines PL caused by image processing with thefilter FL.

In S410 of FIG. 12, the printing device 100 prints the nozzle pattern NPon the sheet SH2. The printing device 100 is provided with the printhead 110 (see FIGS. 2A and 2B). The print head 110 has a plurality ofnozzles Nz configured to eject ink. The nozzle pattern NP includes aplurality of lines PL that are parallel to each other. Each line PL hasa symmetric shape relative to a corresponding center axis AxP. One linePL is printed by a single nozzle Nz. In S420 of FIG. 12, the readingunit 300 optically reads the nozzle pattern NP and generates scanneddata for an image of the nozzle pattern NP. In S430 the processor 210uses the scanned data to acquire target data, i.e., image data of thenozzle pattern NP. In S440 the processor 210 executes an image processon the target data using the filter FL to generate processed image data.In S450 the processor 210 stores the processed image data in the storagedevice 215. This process reduces noise and suppresses distortion oflines PL in the nozzle pattern NP featured in an image represented bythe processed image data. Hence, the processed image data can be used toreliably identify the condition of the nozzles Nz. In this way, theimage process using the filter FL can improve the quality of scanneddata generated by the reading unit 300, and particularly the quality ofa scanned image of the nozzle pattern NP that has been printed by theprinting device 100. As described in FIGS. 2A and 2B, the nozzle patternNP can be used to check the quality of the nozzles Nz (see FIG. 2A). Thenozzle pattern NP can be printed and visually inspected whenmanufacturing the printing device 100, for example. Here, the methodemployed for reducing manufacturing costs may comprise printing thenozzle pattern NP at low density using inexpensive ink, reading theprinted nozzle pattern NP with the reading unit 300, and determining thequality of the nozzle pattern NP (i.e., the quality of the nozzles Nz)by performing an image process on the scanned image. However, since thecolor of the nozzle pattern NP is light when printed at low density,distinguishing the nozzle pattern NP from the background in the scannedimage may be difficult. Here, brightness correction can be performed onthe scanned image to enhance the nozzle pattern NP. However, sincebrightness correction also enhances patterns in the sheet (fiberpatterns in the paper, for example) and digital noise, determining thequality of the nozzles Nz through the image process may be difficult.Since the filter FL in the present embodiment can reduce such patternsin the sheet and digital noise, the present embodiment can improveaccuracy in determining the quality of nozzles Nz through the imageprocess. Thus, inexpensive ink can be used at low density to greatlyreduce manufacturing costs. Note that steps S460 and S470 of FIG. 12 maybe omitted. The processed image data stored in the storage device 215 inS450 may be used in various processes other than the processes in S460and S470. For example, the processor 210 may display a processed imageon the display unit 240 based on the processed image data. By viewingthe processed image, the operator can easily identify problems in linesPL (and hence, defects in nozzles Nz).

In the present embodiment, the image process is performed using thefilter FL in place of the learning model 235. Hence, the quantity ofcomputations required for image processing can be greatly reduced.

B. Second Embodiment

FIG. 13 is a block diagram illustrating another embodiment, i.e., asecond embodiment of the learning model 235 d. The learning model 235 dof the second embodiment is an artificial neural network model referredto as an autoencoder. In this embodiment, the learning model 235 d usesinputted image data to generate image data with reduced noise. Thelearning model 235 d can be used in place of the learning model 235described in the first embodiment.

The learning model 235 d is provided with a first encoder 411, a secondencoder 412, a third encoder 413, a third decoder 415, a second decoder416, and a first decoder 417, for example. These processing units 411through 413 and 415 through 417 are connected in the order given. Theencoders 411 through 413 extract a feature of the image to generate alatent variable 414 representing the extracted feature. Each of theencoders 411 through 413 is configured of a convolutional layer, and apooling layer that processes data outputted from the convolutionallayer, for example. The decoders 415 through 417 generate image data byreconstructing the feature represented by the latent variable 414. Eachof the decoders 415 through 417 is configured of a convolutional layer,and an upsampling layer that processes data outputted from theconvolutional layer, for example.

The noisy pattern image and the reduced-noise pattern image that aregenerated in the example of FIG. 4 are used for training the learningmodel 235 d. Training is performed according to the process described inthe first embodiment with reference to FIG. 10A. The followingdescription will assume that the multifunction peripheral 200 (seeFIG. 1) executes this training. Note that training may be executed byanother data processing device having a high computing power.

In S352 of FIG. 13, the processor 210 inputs noisy pattern image data850 d into the learning model 235 d. In S354 the processor 210 generatesoutput image data 870 d by performing operations with the learning model235 d. In S356 the processor 210 calculates an error value 890 daccording to the loss function LF (see FIG. 10B) based on reduced-noisepattern image data 860 d and the output image data 870 d. Note that thereduced-noise pattern image data 860 d is data corresponding to thenoisy pattern image data 850 d. In S358 the processor 210 adjusts theoperation parameters in the learning model 235 d according to apredetermined algorithm so that the error value 890 d is reduced. Theadjusted operation parameters include the weights and biases of theconvolution filters in the processing units 411 through 413 and 415through 417. The predetermined algorithm may be one that uses thebackpropagation method and gradient descent method, for example. Theprocessor 210 repeatedly performs the steps S352 through S358 describedabove. Once the condition for completing training has been met, in S360the processor 210 stores the learning model 235 d after training in thenonvolatile storage device 230.

The learning model 235 d after training is used in a process forreducing noise in image data. The processor 210 inputs scanned data 750d of the nozzle pattern NP (see FIG. 13) into the learning model 235 d.By performing an operation on the data with the learning model 235 d,the processor 210 can generate output data 770 d of the image withreduced noise.

C. Third Embodiment

FIG. 14 is a flowchart illustrating steps in a process according to athird embodiment for generating training image data (training image datagenerating process). This process differs from the process in FIG. 4according to the first embodiment in that an image of a character isgenerated in place of an image of the nozzle pattern NP. S105 isidentical to the same step in FIG. 4. In S110 a the processor 210selects a target character from among characters to be processed. In thepresent embodiment, the target character is identified by a combinationof a character and typeface. The character is selected from apredetermined character set (the set of alphabetic and numericcharacters, for example). The typeface is selected from a plurality ofpredetermined typefaces (normal, boldface, and italics, for example). InS135 a the processor 210 generates character image data for thecharacter image, i.e., the image of the target character (characterimage generating process). The luminance values for pixels representingthe character are set to 0, and the luminance values for pixelsrepresenting the background are set to 255. In S150 a the processor 210executes a smoothing process on the character image data, as in S150 ofFIG. 4. FIGS. 15A through 15D are explanatory diagrams of the trainingimage data. FIGS. 15A and 15B illustrate character images 810 e and 810f. In the present embodiment, the character images 810 e and 810 f areimages of single characters. The size of the character images 810 e and810 f is predetermined.

In S155 a of FIG. 14, the processor 210 generates first background imagedata (background image generating process), as in S155 of FIG. 4. Thefirst background image has the same size as the character image. FIGS.15A and 15B respectively illustrate first background images 830 e and830 f. In S160 a the processor 210 generates noisy character image data(noisy character image generating process), as in S160 of FIG. 4. FIGS.15A and 15B respectively illustrate noisy character images 850 e and 850f. The luminance value Pn(x, y) for the noisy character images 850 e and850 f are calculated according to the formula shown in FIG. 8D using theluminance value Pt(x, y) for the corresponding character images 810 eand 810 f and the luminance value Pu(x, y) for the corresponding firstbackground images 830 e and 830 f. In S165 a the processor 210 storesthe noisy character image data generated in S160 a in the storage device215 (the nonvolatile storage device 230 in the present embodiment).

In S170 a the processor 210 identifies a representative color value inthe first background image, as in S170 of FIG. 4. In S175 a theprocessor 210 generates reduced-noise character image data(reduced-noise character image generating process), as in S175 of FIG.4. FIGS. 15C and 15D respectively illustrate the character images 810 eand 810 f, second background images 830 em and 830 fm represented by therepresentative color value, and reduced-noise character images 851 e and851 f. The luminance value Pc(x, y) for the reduced-noise characterimages 851 e and 851 f is calculated according to the formula shown inFIG. 9D using the luminance value Pt(x, y) for the correspondingcharacter images 810 e and 810 f and the luminance value Pum(x, y) forthe corresponding second background images 830 em and 830 fm.

In S180 a the processor 210 executes a contrast enhancement process onthe reduced-noise character image data, as in S180 of FIG. 4. FIGS. 15Cand 15D respectively illustrate enhanced-contrast reduced-noisecharacter images 860 e and 860 f. In S185 a the processor 210 stores theadjusted reduced-noise character image data with enhanced contrast inthe storage device 215 (the nonvolatile storage device 230 in thepresent embodiment) in association with the corresponding noisycharacter image data.

In S190 a the processor 210 determines whether generation of image datais complete. In the present embodiment, the condition for completion isthat image data has been generated for all combinations of the pluralityof characters and the plurality of typefaces, but the condition forcompletion may be any of various other conditions. When the generationof image data is not complete (S190 a: NO), the processor 210 selects anunprocessed combination in S110 a. Once the generation of image data iscomplete (S190 a: YES), the processor 210 ends the process in FIG. 14.

The noisy character image data and the adjusted reduced-noise characterimage data generated in the third embodiment can be used to trainvarious prediction models, such as the learning model 235 and learningmodel 235 d (see FIGS. 3 and 13). Here, the character image data mayrepresent characters that do not have linear symmetry. An example of acharacter with no linear symmetry is “F”. Hence, the second term Lfiltermay be omitted from the loss function LF (see FIG. 10B).

D. Fourth Embodiment

FIG. 16 is a block diagram of a learning model 235 g according to afourth embodiment. The learning model 235 g according to the fourthembodiment is an artificial neural network model known as aconvolutional neural network. In the present embodiment, the learningmodel 235 g sorts images into a plurality of categories. Specifically,the learning model 235 g sorts images of nozzle patterns NP intostandard nozzle patterns NP and nonstandard nozzle patterns NP.

The learning model 235 g is provided with a first convolutional layer421, a first pooling layer 422, a second convolutional layer 423, asecond pooling layer 424, a first fully connected layer 425, a secondfully connected layer 426, and a third fully connected layer 427, forexample. These processing units 421 through 427 are connected in theorder given. The learning model 235 g is a typical convolutional neuralnetwork model. The learning model 235 g uses the inputted image data togenerate output data OD specifying a probability PA of a standard nozzlepattern NP, and a probability PB of a nonstandard nozzle pattern NP.

A noisy pattern image generated in the process of FIG. 4 according tothe first embodiment is used for training the learning model 235 g. Asdescribed above, a plurality of noisy pattern images includes imagesrepresenting standard nozzle patterns NP and images representingnonstandard nozzle patterns NP. Label data specifying either “standard”or “nonstandard” is associated with each noisy pattern image. This labeldata is used as supervised data indicating the desired value of theoutput data. In the present embodiment, the adjusted reduced-noisepattern image data is not used. Hence, the process for generatingadjusted reduced-noise pattern image data may be omitted. The followingdescription will assume that the multifunction peripheral 200 executestraining. Note that training may be executed by another data processingdevice having a high computing power.

In S372 of FIG. 16, the processor 210 inputs noisy pattern image data850 g into the learning model 235 g. In S374 the processor 210 generatesthe output data OD by performing an operation with the learning model235 g. In S376 the processor 210 uses the output data OD and superviseddata 880 g to calculate an error value 890 g according to apredetermined loss function. The loss function may be any of variousfunctions suitable for the sorting category, such as cross entropyerror. The supervised data 880 g is the label data associated with thenoisy pattern image data 850 g. In S378 the processor 210 adjusts theoperation parameters of the learning model 235 g according to apredetermined algorithm in order to reduce the error value 890 g. Theadjusted operation parameters include weights and biases in theconvolution filters of the convolutional layers 421 and 423 and weightsand biases in the fully connected layers 425, 426, and 427. Thepredetermined algorithm may be one that uses the backpropagation methodand gradient descent method, for example. The processor 210 repeatedlyperforms steps S372 through S378 described above. Once the condition forcompleting training has been met, in S380 the processor 210 stores thelearning model 235 g after training in the nonvolatile storage device230.

The learning model 235 g after training is used for sorting nozzlepatterns NP. The processor 210 inputs scanned data 750 g for the nozzlepattern NP (see FIG. 16) into the learning model 235 g to generate theoutput data OD through the operations of the learning model 235 g. Theprocessor 210 references the probabilities PA and PB in the output dataOD to sort the nozzle pattern NP in the input image into the categorywith the highest probability.

E. Variations of the Embodiments

(1) The training image data generating process may be any of variousprocesses used in place of the processes described above. For example,in S105 of FIGS. 4 and 14, the processor 210 may acquire one set ofscanned data for one sheet. Subsequently, in S155 and S155 a theprocessor 210 may generate first background image data from the set ofscanned data. Further, one set of first background image data may beshared among a plurality of sets of training image data. Further, theweight k1 in FIG. 9D may differ from the weight k1 in FIG. 8D, and theweight k2 in FIG. 9D may differ from the weight k2 in FIG. 8D. In anycase, the weights k1 and k2 in FIG. 9D are preferably larger than 0. Inthe process of FIG. 6A, one or two of the parameters for width,luminance value, and position may be modified from their standardvalues. Hence, one or two of the three parameters may be fixed at theirstandard values.

Additionally, the smoothing processes in S150 and S150 a may be omittedfrom the corresponding processes in FIGS. 4 and 14. Further, thecontrast enhancement processes of S180 and S180 a may also be omittedfrom the corresponding processes in FIGS. 4 and 14.

(2) The second term Lfilter of the loss function LF (see FIG. 10B) maybe expressed by various functions that calculate a larger value forlarger differences among two weights at linearly symmetrical positions.For example, the second term Lfilter may be the mean square ofdifferences between two weights.

(3) The filter FL (see FIG. 10C) may be any of various sizes, such asFx=7 and Fy=7. Additionally, the horizontal size Fx may differ from thevertical size Fy. Further, in order to improve the symmetry of weights Wrelative to the filter axis of symmetry AxF, the size of the filter FLin the direction orthogonal to the filter axis of symmetry AxF (thehorizontal size Fx in this case) is preferably an odd number.

(4) The test pattern printed by the printing device 100 may be any ofvarious patterns other than the nozzle pattern NP that includes aplurality of lines PL (see FIG. 2A). For example, the test pattern maybe a grid pattern that includes a plurality of vertical lines and aplurality of horizontal lines. The test pattern may include patterns ofvarious shapes, such as circles, ellipses, triangles, and rectangles.The test pattern may include specific parts having shapes that arelinearly symmetric to the axis of symmetry, such as lines, circles,ellipses, triangles, and rectangles. When the loss function LF includesa second term for improving the symmetry of weights W in the filter FL,the loss function LF can suppress distortion (positional deviation, forexample) of specific parts caused by image processing with the filterFL. Further, the size of the filter FL in the direction orthogonal tothe filter axis of symmetry AxF (the horizontal size Fx in the exampleof FIG. 10C) may be the same as or smaller than the size of the specificpart in the direction orthogonal to the axis of symmetry (the width ofthe line PL, for example). The test pattern may include both thespecific part described above (the nozzle pattern NP, for example) and acharacter. The test pattern may include both a figure and a character.Note that the specific part described above may be omitted from the testpattern, in which case the second term may be omitted from the lossfunction LF.

(5) The image data may be expressed by a plurality of color components,such as RGB, CMYK, YCbCr, and the like. The scanned data, input data,output data, training image data (noisy pattern image data,reduced-noise pattern image data, etc.), and target data described abovemay all be RGB bitmap data. If the color value for each pixel in theimage data is expressed as u number of gradation values for u number ofcolor components (where u is an integer greater than or equal to one),the filter FL in FIGS. 3 and 10C includes (Fx×Fy×u) number of weights.This single filter FL may be broken down into u number of subfilterscorresponding to the u number of color components. One subfilter isconfigured of (Fx×Fy) number of weights and is applied to gradationvalues of the corresponding color component. Here, the second termLfilter of the loss function LF (see FIG. 10B) is preferably an overallevaluation value indicating the degree of symmetry for each of the unumber of subfilters. For example, the formula for the second termLfilter shown in FIG. 10B may be used as the formula for the evaluationvalue of one subfilter. Various values identified using the u number ofevaluation values of the u number of subfilters (a representative valuesuch as the mean or median, or the sum) may be used as the overallevaluation value of the u number of subfilters.

(6) The prediction model may have various other configurations than theconfigurations of the learning models 235, 235 d, and 235 g in FIGS. 3,13, and 16. The prediction model is not limited to a model for reducingnoise in image data or sorting image data, but may be a model thatperforms various other processes. The prediction model may include oneor more models selected arbitrary from a group of artificial neuralnetworks, hidden Markov models, and inference engines.

(7) The device used to optically scan images may be any of variousreading devices, such as a digital camera, instead of the reading unit300.

(8) Each of the various data processes, including the training imagedata generating process, the training process for the prediction model,and the image process using a filter may be executed on separate dataprocessing devices. The data processing devices may be various devicesother than the multifunction peripheral 200, such as a personalcomputer, a digital camera, a scanner, or a smartphone. Further, aplurality of devices that can communicate over a network (computers, forexample) may each implement some of the functions in the data processperformed by the data processing device so that the devices as a wholecan provide the functions required for the data process. Here, thesystem comprising these devices corresponds to the data processingdevice of the present disclosure.

Part of the configuration implemented in hardware in the embodimentsdescribed above may be replaced with software and, conversely, all orpart of the configuration implemented in software may be replaced withhardware. For example, the learning models 235, 235 d, and 235 g (seeFIGS. 3, 13, and 16) may be implemented with hardware circuits, such asan application-specific integrated circuit (ASIC), rather than a programmodule.

When all or some of the functions of the present disclosure areimplemented with computer programs, the programs may be stored on acomputer-readable storage medium (a non-transitory storage medium, forexample). The programs may be used on the same storage medium on whichthey were supplied or may be transferred to a different storage medium(a computer-readable storage medium). The “computer-readable storagemedium” may be a portable storage medium, such as a memory card orCD-ROM; an internal storage device built into the computer, such as anyof various ROM or the like; or an external storage device, such as ahard disk drive, connected to the computer.

While the description has been made in detail with reference to specificembodiments and variations, it would be apparent to those skilled in theart that various changes and modifications may be made thereto.

What is claimed is:
 1. A method for generating an image processingfilter for image processing, the method comprising: adjusting aconvolution filter using a neural network to acquire an adjustedconvolution filter, the convolution filter including a plurality ofweights corresponding to respective ones of a plurality of pixels withina filter region, the neural network being configured of a convolutionallayer configured to perform convolution using the convolution filter,the adjusting comprising: inputting first training image data into theneural network to generate output image data; calculating an evaluationvalue based on a loss function using the output image data and secondtraining image data; and adjusting the plurality of weights torespective ones of a plurality of adjusted weights so as to reduce theevaluation value, the adjusted convolution filter including theplurality of adjusted weights; and extracting data from the adjustedconvolution filter as data for the image processing filter, wherein thefirst training image data represents a first training image includingnoise, the first training image reproducing a test pattern including aspecific part having a shape linearly symmetric to an axis of symmetry,wherein the second training image data represents a second trainingimage including reduced noise compared to the first training image, thesecond training image reproducing the test pattern, and wherein the lossfunction includes a first term and a second term, the first termspecifying a magnitude of a difference between the output image data andthe second training image data, the second term growing smaller assymmetry of the plurality of weights relative to a filter axis ofsymmetry in the filter region increases, the filter axis of symmetrybeing parallel to the axis of symmetry.
 2. The method according to claim1, wherein the second training image has more enhanced contrast than thefirst training image.
 3. The method according to claim 1, wherein thesecond term is smaller for a smaller difference between a first weightand a second weight included in the plurality of weights, the firstweight and the second weight being arranged in the filter region atsymmetric positions about the filter axis of symmetry.
 4. An imageprocessing method using an image processing filter comprising:generating data for the image processing filter in accordance with themethod according to claim 1; acquiring target data representing a nozzlepattern by optically reading the nozzle pattern printed on a sheet, thenozzle pattern being printed on the sheet by a printing device having aprint head, the print head having a plurality of nozzles each of whichconfigured to eject ink, the nozzle pattern including a plurality oflines, the plurality of lines being parallel to each other, each of theplurality of lines having a center axis and having a symmetric shaperelative to the center axis, each of the plurality of lines beingprinted by one of the plurality of nozzles; generating processed imagedata by executing an image process on the target data using the imageprocessing filter; and storing the processed image data in a storagedevice.
 5. A method for generating training image data to be used fortraining a prediction model, comprising: (a) generating first trainingimage data representing a first training image formed by arranging anobject image of an object on a first background image, the firstbackground image being at least a portion of a sheet image acquired byoptically reading a sheet to be used for printing; and (b) storing thefirst training image data in a storing device, wherein the (a)generating comprises: (a1) setting a color value for a first objectpixel within an object region of the first training image in which theobject image is arranged, the color value for the first object pixelbeing set to a sum of a first value and a second value, the first valuebeing obtained by multiplying a color value for a pixel corresponding tothe first object pixel in the object image by a first object coefficientgreater than zero, the second value being obtained by multiplying acolor value for a pixel corresponding to the first object pixel in thefirst background image by a first background coefficient greater thanzero; and (a2) setting a color value for a first background pixel withina nonobject region of the first training image in which the object imageis not arranged, the color value for the first background pixel beingset to a value obtained by multiplying a color value for a pixelcorresponding to the first background pixel in the first backgroundimage by a coefficient greater than the first background coefficient. 6.The method according to claim 5, further comprising: (c) identifying arepresentative color value in the first background image, a plurality ofcolor values for respective ones of a plurality of first backgroundpixels being set in the (a1) setting, the representative color valuerepresenting at least part of the plurality of color values forrespective ones of the plurality of first background pixels; (d)generating second training image data representing a second trainingimage formed by arranging the object image on a second background imagerepresented by the representative color value; and (e) storing thesecond training image data in the storing device, wherein the (d)generating comprises: (d1) setting a color value for a second objectpixel within an object region of the second training image in which theobject image is arranged, the color value for the second object pixelbeing set to a sum of a third value and a fourth value, the third valuebeing obtained by multiplying a color value for a pixel corresponding tothe second object pixel in the object image by a second objectcoefficient greater than zero, the second value being obtained bymultiplying the representative color value by a second backgroundcoefficient greater than zero; and (d2) setting a color value for asecond background pixel within a nonobject region of the second trainingimage in which the object image is not arranged, the color value for thesecond background pixel being set to a value obtained by multiplying therepresentative color value by a coefficient greater than the secondbackground coefficient.
 7. The method according to claim 6, wherein therepresentative color value is an average value of the at least part ofthe plurality of color values for respective ones of the plurality offirst background pixels.
 8. The method according to claim 5, wherein theobject includes of lines aligned in a first direction, the M number oflines being juxtaposed in a second direction orthogonal to the firstdirection, where M is an integer greater than or equal to two.
 9. Themethod according to claim 8, further comprising: (f) generating Q numberof sets of first training image data representing respective ones of Qnumber of first training images by performing the (a) generating Qnumber of times, where Q in an integer greater than or equal to two,wherein each of the Q number of first training images includes theobject image of the object including the M number of lines, the M numberof lines including at least one nonstandard line different from astandard line, the standard line having a standard luminance and astandard line width and being positioned at a standard position in thesecond direction, the standard position of the standard line beingassociated with an order of the standard line among the M number oflines arranged in the second direction, each of the at least onenonstandard line having a luminance and a line width and beingpositioned at a position in the second position, at least one of theluminance, the line width, and the position being different fromrespective ones of the standard luminance, the standard line width, andthe standard position.
 10. The method according to claim 9, wherein theQ number of sets of first training image includes: at least one firsttraining image including at least one nonstandard line each of which hasthe luminance different from the standard luminance; at least one firsttraining image including at least one nonstandard line each of which ispositioned at a position different from the standard position in thesecond direction; and at least one first training image including atleast one nonstandard line each of which has a line width different fromthe standard line width.
 11. The method according to claim 9, wherein atotal number of the at least one nonstandard line included in each ofthe Q number of sets of first training image is different from eachother.
 12. The method according to claim 5, wherein the object includesa character.
 13. A non-transitory computer readable storage mediumstoring a prediction model installed on and executed by a computer, theprediction model, when executed by the computer, causing the computer toperform generating relevant data based on image data representing asheet image, the sheet image being generated by optically reading asheet on which an object is printed, the relevant data being data aboutthe object, wherein the prediction model has been trained using trainingimage data representing a training image formed by arranging an objectimage of an object on a background image, the background image being atleast a portion of a sheet image acquired by optically reading a sheetto be used for printing, and wherein the training image data isgenerated according to a method including: setting a color value for anobject pixel within an object region of the training image in which theobject image is arranged, the color value for the object pixel being setto a sum of a first value and a second value, the first value beingobtained by multiplying a color value for a pixel corresponding to theobject pixel in the object image by an object coefficient greater thanzero, the second value being obtained by multiplying a color value for apixel corresponding to the object pixel in the background image by abackground coefficient greater than zero; and setting a color value fora background pixel within a nonobject region of the training image inwhich the object image is not arranged, the color value for thebackground pixel being set to a value obtained by multiplying a colorvalue for a pixel corresponding to the background pixel in thebackground image by a coefficient greater than the backgroundcoefficient.